Automation for the people: Continuous testing

Developer tests, which when run often in a Continuous Integration (CI) environment, effectively act like a spot light on code quality. This is because these tests, if written effectively, can find issues like defects almost as soon as they're created. Tests that aren't run often are less effective because the time between when a defect is coded and when it can be discovered is long, but running tests continually (that is, every time code changes) ensures that unintended behavior is discovered quickly.

This article covers the following:

• Running JUnit tests via Ant
• Executing longer running component tests using JUnit and DbUnit
• Determining which methods take too long to execute using JUnitPerf
• Running Web-based functional tests with Selenium
• Assessing code coverage with Cobertura
• Testing continually with CruiseControl

I provide an overview of the different types of developer tests along with examples that you can add to your build process and run continually using a Continuous Integration system.

Unit testing a la JUnit

Sometimes I hear developers lump the term developer tests as simple unit tests; however, I've found that it's helpful to refine the term unit testing as something more specific. To me, unit tests are fast running tests that typically test individual classes that don't have heavy external dependencies like databases. For example, Listing 1 defines a unit test that uses JUnit to verify a stubbed out data class called BeerDaoStub. The technique of testing against an interface that doesn't actually connect to a database, for example, is an approach that you can use to verify business functionality without incurring expensive setup costs. Plus, doing so keeps the test as a true unit test.

public void setUp() {    beerService = new BeerDaoStub();  }    public void testUnitGetBeer() {    Collection beers = beerService.findAll();    assertTrue(beers != null && beers.size() > 0);  }  

Once you've written a few unit tests, you can always run them through an IDE, but you'll also want to run them as part of a build process. Ensuring that tests run successfully through a build process means these same tests can be kicked off in the context of a CI build too.

Listing 2 is a snippet of an Ant script that demonstrates the junit task to execute a batch of unit tests. The beauty with this task, working in concert with JUnit, is that any tests that I've defined are now automatically run and if any of these tests fail, the build will fail also -- by using the haltonfailure attribute.

haltonfailure="true" dir="${basedir}" printsummary="yes">                                                  

Notice that test.unit.dir designates the location of the tests. This is an effective way to segregate these tests (unit, in this case) from other tests. By utilizing this technique, you can run faster tests first, followed by slower tests (such as component, functional, and system tests) by defining additional Ant targets.

Back to top

Collecting component tests

Because unit tests execute so quickly, they are easy to run often as a part of a build. These tests, however, don't achieve a high rate of code coverage -- their isolated nature means they only test a portion of the functionality. Writing tests that reach more code and therefore exercise more functionality usually takes more legwork in the form of adjunct frameworks. Once you start to use these helper frameworks to write tests, these tests start to become higher level ones that I tend to categorize as component tests.

Component tests are essentially tests that verify more than one class and, typically, rely on external dependencies such as a database. Component tests are written in much the same way as unit tests, except that instead of mocking or stubbing classes to force isolation, these tests bite the bullet, so to speak, and enlist frameworks to facilitate working with external dependencies. For example, I often employ the DbUnit framework to help manage a database so that my component tests can verify the functionality of code that relies on the database's data.

Controlling database state with DbUnit

DbUnit is a framework that makes the process of testing against a database much simpler. It provides a standard XML format for defining test data that can be used to select, update, insert, and remove data from a database. Keep in mind, DbUnit doesn't replace a database; it provides a more efficient mechanism for handling test data. With DbUnit, you can write tests that rely on specific data, which DbUnit ensures is present in an underlying database.

You can use DbUnit programmatically in JUnit or you can utilize it as part of a build process. The framework comes with an Ant task that provides a way to manipulate, export, or compare data in a database using XML files. For example, Listing 3 demonstrates the dbunit task, which, in my case, inserts test data into my target database and then removes the data after having run all my component tests:

              <dbunit driver="com.mysql.jdbc.Driver"        url="jdbc:mysql://localhost:3306/brewery"        userid="${db.username.system}"  	  classpathref="db.lib.path"        password="${db.password.system}">                                                                                                                                                                                                      

As you can see in Listing 3, my component tests can now rely on specific data residing in the database during their execution. What's more, this process is now repeatable because I've removed all the data after all tests successfully execute.

Seeding the database

You can use the INSERT and DELETE operation types for the dbunit task in conjunction with a seed file that contains XML elements representing database tables and associated rows. For example, Listing 4 is the contents of the seedFile.xml file referenced in Listing 3. Each BEER element represents a database table also named BEER and each of the BEER element's attributes and their values map to corresponding database column names and values.

As you may have noticed from Listing 3, you can reuse DbUnit's seed files for different operations. In my case, I used the file in listing 4 to seed the database before running my component tests and then used the same file to indicate which data to delete from the database upon test completion.

Back to top

Partaking in performance tests

Performance testing is often performed (no pun intended) long after developers have finished coding -- yet it's often the case that performance issues could have been found (and most likely solved) much earlier in the development cycle. Luckily, there is a way to solve this problem: continuous testing, or more specifically, continuously running JUnitPerf tests.

JUnitPerf is perfect for performance testing

JUnitPerf is a framework that works in concert with JUnit to execute test cases within a predefined time limit: If a method under test takes longer than a desired threshold, the test is considered a failure. By integrating performance tests into an automated build, you can effectively monitor application performance and even fail a build if performance issues pop up.

I prefer to use JUnitPerf as a simple way to discover early performance problems rather than as a mechanism to measure execution time though; tools like profilers are much more capable at providing this type of measurement. In essence, you can think of JUnitPerf as an early warning system.

In Listing 5, I define a JUnit test that uses JUnitPerf to verify the execution time of a test called testLongRunningMethod in the BeerServicePerformanceTest test class. If the test method happens to take longer than 1000 milliseconds to execute, it fails.

package com.beer.business.service;  import com.clarkware.junitperf.*;  import junit.framework.Test;    public class ExampleTimedTest {    public static Test suite() {              long maxElapsedTime = 1000;              Test testCase = new BeerServicePerformanceTest("testLongRunningMethod");      Test timedTest = new TimedTest(testCase, maxElapsedTime);              return timedTest;    }  	    public static void main(String[] args) {      junit.textui.TestRunner.run(suite());    }    }  

Be careful in using precise timings as a gauge for the execution time of a method; the setup and tear down time of a test is included in the overall execution time. Plus, precise measuring of execution speed is more of an art form than a science when it comes to early performance testing.

Back to top

Functional tests with Selenium

You can write all of the unit and component tests you want, but if you're writing an application that provides a user interface of some type (like a Web application, for example), you'll need to test the presentation layer. In the case of a Web application, you'll want to verify that the navigation of a user scenario and additionally verify that the functionality of the scenario pans out. Until recently, however, this sort of testing has proven to be a burden as typically, you'd need to purchase tools that promoted late cycle testing. What's more, these tools rarely fit into a build process even if the tests were built early enough.

Enter Selenium

In the last few years, however, a number of open source tools have sprung up that address functional testing; moreover, you can easily employ these tools early in the development life cycle. Tools such as Selenium and Watir are open source; what's more, they have been built with developers in mind. Besides programmatically defining Selenium tests in various languages (like Java� programming and Python, for example), Selenium also provides an easy to learn table-driven format that can be used by non-technical types as well.

The Selenium framework uses JavaScript to execute Web-based acceptance tests that open a browser and run table-driven tests. For example, Listing 6 demonstrates an HTML table that represents a simple Selenium test. The various steps of the test open a Web application and then perform a login with a valid username and password. The test's results are generated in an HTML table that can be viewed after Selenium completes running all the tests.

            
  
  
  
    
open
    
/beer/
    
  
  
    
type
    
username
    
admin
  
  
    
type
    
password
    
password
  
  
    
clickAndWait
    
//input[@value='Login']
    
  
  
    
verifyTextPresent
    
Logged in as admin
    
  
  
      

You can define multiple acceptance tests using the table-based format in Listing 6. You can also group tests into suites and execute an entire suite at a time.

Driving Selenium through Ant

What's great about Selenium is that it was created from the ground up with CI in mind because you can run Selenium tests from build tools like Ant. What's more, because of the forward thinking of the framework's designers, if any Selenium acceptance tests fail, you can also fail the entire build. For example, Listing 7 demonstrates an Ant task that uses the Selenium Remote Control server to execute a series of table-driven tests against a Web application:

When executing Selenium tests, don't be alarmed when the framework opens a Web browser, executes your tests at lightning speed, and then closes the browser and generates an HTML report -- it's one more quick and easy way you can learn about problems early in the development life cycle (when they are easier to fix).

Back to top

Code coverage with Cobertura

Now that you've written a bunch of tests, how do you determine what all those tests execute? Luckily, answering this question is where a code coverage tool like Cobertura shines. A code coverage tool reports test coverage -- either in the form of line or branch coverage -- that represents the amount of code that is being touched when a test is run.

Listing 8 demonstrates an Ant script that uses Cobertura to generate an HTML report of the code coverage achieved by running a series of JUnit tests:

Cobertura produces an HTML report like the one in Figure 1. Notice the line and branch coverage percentages by package. You can click on each package to get class level line and path percentages and even see which source code lines were executed and how many times they was run.

Figure 1. HTML report produced using Cobertura and Ant

You've seen various types of tests and even how to measure the coverage of those tests -- but how do you ensure that the execution of these tests happens at regular intervals? As it happens, this is where a CI server -- like CruiseControl steps up to the plate, as I'll show you next.

Running tests continually

Once you've incorporated the execution of these various developer test types into a build process, you can run some (or all) of these tests as part of a CI process. For example, in Listing 9, which is a snippet of CruiseControl's config.xml file, I define a number of things. First, I have CruiseControl monitoring a Subversion repository for any changes every two minutes. If it discovers any changes have occurred, CruiseControl kicks off a delegating build script (typically, you will see this written in Ant) called build-${project.name}.xml. The delegating build script calls the project's build script, which executes a compile and runs any tests.

I've also defined some logic to merge the results of all my different types of tests into a CruiseControl log file. Furthermore, I'm also using CruiseControl's capability to link (using the artifactspublisher tag) the reports generated by different tools into the Build Artifacts link, which is available from the CruiseControl dashboard application.

...                                              <artifactspublisher       dir="projects/${project.name}/_reports/"       dest="projects/artifacts/${project.name}"/>    ...  

You won't necessarily run every test defined with every source change applied to your version control repository. For instance, you can set up your CI system to execute a build that only runs unit tests during code check-ins (which is often referred to as a commit build). You can complement commit builds with more heavy-weight style builds that run component, functional, performance test and even code inspections, for example (Resources). These builds can be run more infrequently (like once a day, for instance). Alternatively, you can run these tests and inspections immediately after your commit build.